Change-Resilient Design and Dataflow Optimization for Distributed XML Stream Processors

We propose a new stream-processing framework based on a virtual assembly line (val) model. We instantiate the val framework obtaining ∆-XML, an approach for designing and optimizing distributed XML processing pipelines. val/∆-XML greatly simplifies the design of change-resilient dataflow pipelines: XML processors (called actors) can be inserted, deleted, and their “scope of work” (the parts of the stream they can read from and write to) changed freely, without compromising the overall process pipeline design. Unlike conventional approaches that rely on adapters to “glue” together processing components, our actors employ flexible configurations that select only relevant portions of the input stream. ∆-XML pipelines are not only more flexible and change resilient than current approaches, but can also be optimized by compiling them into dataflow process networks that minimize shipping cost in distributed settings: Using a static type inference approach based on regular expression types for XML, we show how to perform a dataflow analysis to determine XML stream fragments that are relevant to an actor, allowing irrelevant fragments to be bypassed (“shipped”) to downstream actors. We also show that our approach is optimal for distributed XML pipelines, given the type information available in ∆-XML.